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Abstract. In this paper, we investigate a nonparametric approach to provide a re- 
cursive estimator of the transition density of a non-stationary piecewise-deterministic 
Markov process, from only one observation of the path within a long time. In this frame- 
work, we do not observe a Markov chain with transition kernel of interest. Fortunately, 
one may write the transition density of interest as the ratio of the invariant distributions 
^ of two embedded chains of the process. Our method consists in estimating these invariant 

measures. We state a result of consistency under some general assumptions about the 
main features of the process. A simulation study illustrates the well asymptotic behavior 
of our estimator. 

m 
H 

Xjy 1. Introduction 

The purpose of this paper is to investigate a nonparametric recursive method for esti- 
mating the transition kernel of a non-stationary piecewise-deterministic Markov process, 
i ** H i from only one observation of the process within a long time interval. 

— Piecewise-deterministic Markov processes (PDMP's) have been introduced in the litera- 

ture by Davis in [7]. They are a general class of non-diffusion stochastic models involving 
deterministic motion broken up by random jumps, which occur either when the flow reaches 
ly-) the boundary of the state space or in a Poisson-like fashion. The path depends on three 

local features namely the flow <3>, which controls the deterministic trajectories, the jump 
— , rate A, which governs the inter-jumping times, and the transition kernel Q, which deter- 

mines the post-jump locations. An appropriate choice of the state space and the main 
characteristics of the process covers a large variety of stochastic models covering problems 
in reliability (see [7] and [5]) or in biology (see [13] and [JO]) for instance. In this context, 
. £^ it appears natural to propose some nonparametric methods for estimating both the char- 

^ acteristics A and Q, which control the randomness of the motion. Indeed, the deterministic 

flow is given by physical equations or deterministic biological models. In [2], Aza'is et al. 
proposed a kernel method for estimating the conditional probability density function asso- 
ciated with the jump rate A, for a non-stationary PDMP defined on a general metric space. 
This work was based on a generalization of Aalen's multiplicative intensity model and a 
discretization of the state space. In the present paper, we assume that Q admits a density 
q with respect to the Lebesgue measure, and we focus on the nonparametric estimation 
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of this function, from the observation of a PDMP within a long time, without assumption 
of stationarity. In addition, since measured data are often processed sequentially, it is 
convenient to propose a recursive estimator. 

Nonparametric estimation methods for stationary Markov chains have been extensively 
investigated, beginning with Roussas in [20]. He studied kernel methods for estimating 
the stationary density and the transition kernel of a Markov chain satisfying the strong 
Doeblin's condition. Later, Rosenblatt proposed in [19] some results on the bias and the 
variance of this estimator in a weaker framework. Next, Yakowitz improved in [22] the 
previous asymptotic normality result assuming a Harris-condition. Masry and Gyorfi in 
[17], and Basu and Sahoo in [3], have completed this survey. There exists also an extensive 
literature on nonparametric estimates for non-stationary Markov processes. We do not 
attempt to present an exhaustive survey on this topic, but refer the interested reader 
to [6, 8, 9, 11, 14, 15, 16] and the references therein. In this new framework, Doukhan 
and Ghindes have investigated in [8] a bound of the integrated risks for their estimate. 
Hernandez-Lerma et al. in [11] and Duflo in [9] made inquiries about recursive methods for 
estimating the transition kernel or the invariant distribution of a Markov chain. Liebscher 
gave in [16] some results under a weaker condition than Doeblin's assumption. More 
recently, Clemencon in [6] proposed a quotient estimator using wavelets and provided the 
lower bound of the minimax L p -risk. Lacour suggested in [J 5] an estimator by projection 
with model selection, next she introduced in [ ] an original estimate by inquiring into a 
new contrast derived from regression framework. 

Our investigation and the studies of the literature mentioned before are different and 
complementary. In this paper, we propose to estimate the transition density q of a PDMP 
by kernel methods. Nevertheless, we do not observe a Markov chain whose transition 
distribution is given by q. Fortunately, one may write the function of interest as the ratio 
of two invariant measures: the one of the two components pre-jump location and post- 
jump location, over the one of the pre-jump location. Indeed, Q(x,A) is defined as the 
conditional probability that the post-jump location is in A, given the path is in x just 
before the jump. Therefore, we suggest to estimate both these invariant measures in order 
to provide an estimator of the transition kernel Q. A major stumbling block for estimating 
the invariant law of the pre-jump location is related with the transition kernel of this 
Markov chain, which may charge the boundary of the state space. As a consequence, the 
transition kernel, as well as the corresponding invariant distribution, admits a density only 
on the interior of the state space. The investigated approach for estimating the invariant 
measure is based on this property of the transition kernel. But the main difficulty appears 
for analyzing the two-components process pre-jump location, post-jump location. This 
Markov chain has a special structure, because its invariant distribution admits a density 
function on the interior of the state space, unlike its transition kernel. Indeed, the pre-jump 
location is distributed on the curve governed by the deterministic flow initialized by the 
previous post-jump location. As a consequence, the author have to explore a new method 
for estimating the two-dimensional invariant measure of interest. The proposed one is more 
universal, but implies a more restrictive assumption on the shape of the bandwidth. 

An intrinsic complication throughout the paper comes from the presence of deterministic 
jumps, when the path tries to cross the boundary of the state space. Indeed, this induces 
that the invariant distributions mentioned above may charge a subset with null Lebesgue 
measure. This important feature has been introduced by Davis in [7] and is very attractive 
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for the modeling of a large number of applications. For instance, one may find in [ ] an 
example of shock models with failures of threshold type. One may also refer the reader 
to [13], where the authors develop a PDMP to capture the mechanism behind antibiotic 
released by a bacteria. Forced jumps are used to model a deterministic switching when the 
concentration of nutrients rises over a certain threshold. 

The paper is organized as follows. Section 2 is devoted to the precise formulation of 
our problem. The sequel of the article is divided into three parts. Section 3 is devoted 
to a recursive kernel estimator of the invariant measure of the pre-jump location. More 
precisely, in Subsection 3.1, we examine the ergodicity of the Markov chain of the pre-jump 
locations. Next, in Subsection 3.2, we introduce a nonparametric estimator of the corre- 
sponding invariant distribution and we prove its almost sure convergence in Proposition 
3.11. In Section 4, we investigate a nonparametric recursive method for estimating the 
invariant distribution of the two-components chain pre-jump location, post-jump location. 
First, some properties of this Markov process are studied in Subsection 4.1. The recursive 
estimator is provided in Subsection 4.2 and the associated convergence is stated in Propo- 
sition 4.7. Our nonparametric estimator of the function of interest q is defined as the ratio 
of both the previously mentioned recursive estimators. The main result of consistency lies 
in Theorem 4.8. Finally, Section 5 deals with numerical considerations for illustrating the 
asymptotic behavior of our estimate. 

2. PRELIMINARIES 

This section is devoted to the definition of a piecewise-deterministic Markov process on 
R d , where d is an integer greater or equal to 1. The process evolves in an open subset E 
of H d equipped with the Euclidean norm | • |. The motion is defined by the three local 
characteristics (A, Q,&). 

• <3? : H d x R — > H d is the deterministic flow. It satisfies, 

V£ G R d , Vs, t E R, * € (t + s) = * 4e(t )(s). 
For each £ G E, denotes the deterministic exit time from E: 

t+(£) = inf{t > : dE}, 

with the usual convention inf = +oo. 

• A : H d —J- R+ is the jump rate. It is a measurable function which satisfies, 

V£ G R d , 3e > 0, f A(* £ (s))ds < +00. 
Jo 

• Q is a Markov kernel on (R, d ,B(R d )) which satisfies, 

V£e£, Qfc,E\{Z}) = l and Q(£,tf) = l. 

There exists a filtered probability space (O, A, (.Ft), P), on which the process (Xt) is defined 
(see [7]). Starting from x G E, the motion can be described as follows. T\ is a positive 
random variable whose survival function is, 

Vt > 0, P(Ti > t\X = x)= exp (- J \($ x (s))ds) l { o< t<t+(x)} . 
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One chooses an ^-valued random variable Z\ according to the distribution Q(<& X (T\), ■) 
Let us remark that the post-jump location depends only on the pre-jump location <& x (Ti) 
The trajectory between the times and T\ is given by 

' for < t < Ti, 

Z\ for t = T\. 



Now, starting from Xj\ , one selects the time S2 = T2 — T% and the post-jump location Z2 
in a similar way as before, and so on. This gives a strong Markov process with the T^'s as 
the jump times (with To = 0). One often considers the embedded Markov chain (Z n ,S n ) 
associated to the process (Xt) with Z n = Xt„, S n = T n — T n _i and So = 0. The Z^s 
denote the post-jump locations of the process, and the S n 's denote the interarrival times. 

In this paper, we assume that the transition kernel Q admits a density q. Our main 
objective is the estimation of this function from the observation of one trajectory of the 
process within a long time. 

Assumption 2.1. We assume that the transition kernel Q admits a density q. That is, 

V£ G R d , VB € B(R d ), Q(Z,B) = [ q(£,z)dz. 

Jb 

In addition, q is assumed to be piecewise- continuous. 

In the following, we shall consider the discrete-time process (Z~) defined by, 

Vn>l, Z- = $ Zn _ 1 (S n ). 

This sequence is naturally of interest. Indeed, the transition kernel Q describes the tran- 
sition from Z~ to Z n . Z~ stands for the location of (Xt) just before the nth jump. We 
shall prove that (Z~) is a Markov chain in Lemma 3.1. 

Our main objective in this paper is to provide a recursive estimator of q(x,y). The 
recursive estimator of q(x, y) that we consider may be written as follows, 



ln(x,y) 



where Wj = wij~^, Vj = vij~ a , with a, j3 > 0, and K is a kernel function satisfying 
Assumptions 3.9. Under some assumptions imposed in the sequel, we will state our major 
result of consistency in Theorem 4.8, that is 

q n (x, y) —> q(x, y), as n goes to infinity. 

Let us introduce some notations. In all the sequel, / and G denote the conditional 
probability density function and the conditional survival function associated with A(<3?.(-)). 
Precisely, for all z E R d and t > 0, 

'■t 





G(z,t) = exp(-^ t A(^( S ))d^ , 
f(z,t) = \($ z (t))G(z,t). 
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In addition, throughout the paper, S denotes the conditional distribution of S n +\ given 
Z n , for all integer n. For all z E E and T E £>(R+), we have 



The relation between S and the conditional survival function G is given, for all z E R 
and t > 0, by G(z,t) = S(z,]t,+oo[). 

In the next section, we focus on the invariant distribution of the process (Z~) and we 
investigate a nonpar ametric way to estimate it. 



The main objective of this section is the estimation of the invariant distribution of 
the Markov chain (Z~). This section is divided into two parts. In the first one, we are 
interested in the existence and the uniqueness of the invariant distribution of (Z~), and 
in the properties of its transition kernel 7Z. In the second part, we propose a recursive 
estimator of the invariant distribution of (Z~) and we investigate its asymptotic behavior. 

3.1. Some properties of (Z~). In this part, we focus on the process (Z~), which is a 
Markov chain on E. We especially investigate its transition kernel 1Z and the existence of 
an invariant measure. 



Lemma 3.1. [Z n ) is a Markov chain whose transition kernel 1Z is given, for all y E E 
and B E B(E), by 



(1) 



s(z,r) 



P (S n+ i E T\Z n = z,a(Zi, Si : < i < n)) 
/ f(z,s)ds + l r (t + (z))G(z,t + (z)). 




3. ESTIMATION OF THE INVARIANT DISTRIBUTION OF (Z~) 




where the conditional distribution S has already been defined by (1). 




B[S(z,^ 1 (B)nK + )\Z n = z,Z 1 

%^(B)nR + ). 



n ' 




As a consequence, 



P(Z" +1 E B\ Z, 




P(Z" +1 E B\Z n = z, Z~, . . . , Zi)Q(Z~,dz) 




which provides the result. 



□ 
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We focus on the ergodicity of (Z~) by using Doeblin's condition. First, we consider the 
function m : E —> R+ defined for any z G E by 

(3) m(z) = ia£_q(y,z). 

y&E 

In addition, \i denotes the measure on (E, B{E)) defined by, 

(4) MB G B(E), fi(B) = fi°(B) + ^(B), 
where the measures /jP and p 1 are given by 



p}\B) = / m(z) / f(z,s)ds)dz, 
Je \J<i>- 1 ( J B)n[o,t+( z )[ / 

»\B) = f m(z)l^i {B) (t + (z))G(z,t+(z))dz. 

J E 

/jP may only charge the interior of the state space, while p} may be strictly positive only 
on the boundary of E. By the expression of 1Z (2) and the one of m (3), we have the 
straightforward inequality 

(5) Vy G E, \/B £ B(E), K{y, B) > fi(B). 
The first assumption that we impose concerns the measure \x. 

Assumption 3.2. We assume that the measure fi defined by (4) is a nondegenerate one. 
That is, fj,/ (jl{E) is a probability measure. 

Under this assumption, one may state that the Markov chain (Z~) is ergodic. 
Proposition 3.3. We have the following results. 

• The Markov chain (Z~) is \x-irreducible, aperiodic and admits a unique invariant 
measure, which we denote by ir. 

• There exits p > 1 and k > such that, 

(6) Vn > 1, sup \\TZ n (C, ■) - tt\\tv < np~ n , 

where \\ ■ \\tv stands for the total variation norm. 

• In addition, (Z~) is positive Harris-recurrent. 

Proof. By definition and the inequality (5), (Z~) is ^-irreducible and aperiodic (see [18, 
page 114]). Moreover, the transition kernel 1Z obviously satisfies Doeblin's condition (see 
[18, page 396] for instance), 

H{B) > e =>- K{y,B) > e, 

once again by (5). On the strength of Theorem 16.0.2 of [ ], (Z~) admits a unique 
invariant measure ir since it is aperiodic and (6) holds. In addition, from Theorem 4.3.3 of 
[12], (Z~) is positive Harris-recurrent. □ 

Now, we shall impose some assumptions on the characteristics relative to the flow of the 
process. Under these new constraints, one may provide a more useful expression of 1Z. In 
the sequel, for £ G E, t~ (£) denotes the deterministic exit time from E for the reverse flow, 

t~(0 = sup{t<0 : Q x (t)edE}, 

with the usual convention sup0 = — oo. Remark that t~(£) is a negative number. 



A RECURSIVE NONPARAMETRIC ESTIMATOR FOR THE TRANSITION KERNEL OF A PDMP 7 

Assumptions 3.4. 

(i) The flow $ is assumed to be C 1 -smooth. For any (z, t) £ H d x R 7 D$ z (t) is defined 
by 



(7) D$ z (t) 



det 



'l<i,j<d 



(ii) For any t £ R, (ft ■ R — > R , defined by (ft(x) = $ x (t), is an injective application. 

A useful expression of the transition kernel 7Z is stated in the following proposition. 
Proposition 3.5. Let y £ E and B £ B{E). We have 

(8) K(y,B)= [ r(y,z)dz + K(y,BndE), 

JBnE 

where the conditional density function r is given by, 

(9) VzGE, r(y,z)= [ ' " q(y, s )D$ z (-s)ds. 

Jo 

Proof. Let B C E. First, we fix t > 0. We define the set At by 

A t = {^(-t) : £ £ B}, 

and we examine the function ipt : At —> B defined by <pt(x) = $x(t), x E At. (ft is a 
C 1 -smooth injective application (see Assumptions 3.4). Furthermore, for any z £ B, 

<Pt($z(.-t)) = &*,(-t)(t) = Z, 

with $ 2 (— t) £ At by definition of At. Consequently, (ft is a C 1 -one-to-one correspondence. 
The inverse function ip~[ l is given by ip^ 1 (x) = <fr x (—t), so it is C 1 -smooth too. Thus, (ft is 
a C 1 -diffeomorphism from At into B, which allows us to consider it as a change of variable. 
In particular, this shows the relation 

(10) {zeE,te R+, $ z (t) eB) ^ (t £ R+, z £ E, z e A t ) . 

Moreover, the Jacobian matrix J^-i of the inverse function ip^ 1 satisfies, 



Vx £ K d , J -i(x) 



'd$£\-t) 
dxj 



'l<i,j<d 



By (1) and (2), we have 



TZ(y,B) = [ [ q(y,t)f(z,t)dt. 
je J$r 1 (B)nR+ 

Together with (10), we obtain 

TZ(y, B) = J l E (z)f(z, t)q(y, z)dz^ dt. 

By the change of variable (ft, we have 

n(y,B) = ^l s (^- 1 (O)/(^ 1 (e),09(y>^ 1 (O)|det J^(0|d^dt 

= / (^l E (^H))/(^H),t)g(y,^H))^H)d^dt, 
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where is defined by (7). We remark that 

(<^(-i)e£)^(0<t<-i*(O), 
so f)) = l{o<t<-t*(£)}- By Fubini's theorem, this yields to the expected result. □ 

In the light of this result, one may obtain the following one about the invariant distribution 
7r of the Markov chain (Z~): tt admits a density with respect to the Lebesgue measure in 
the interior of the state space. In addition, one may exhibit a link between this density 
and r. 

Corollary 3.6. There exists a non-negative function p such that 

(11) V-B G B(E), tt(B) = [ p{x)dx + Tr(BndE). 

J bde 

In addition, p is given by the expression, 

(12) Vx G E, p(x) = / ir(dy)r(y, x). 

Je 

Proof, fx is an irreducibility measure for 1Z. As a consequence and according to Theorem 
4.2.2 of [18], the maximal irreducibility measure Jl is equivalent to the measure Jl' given 
for any B G B(E), by 

Ji'(B) = I Kdy)^ K n (y,B)~. 

JE n>0 

1Z admits a density on the interior E of the state space E of the Markov chain (Z~) (see 
Proposition 3.5). As a consequence, this is the case for lZ n , too. Indeed, for any set B 
such that \d{B H E) = 0, we have 

1Z n (y, BHE) = [iZ^iy, dz)Ti{z, B n E) = 0. 
Je 

Therefore, J1'(B fl E) = 0. Finally, 

\ d (BnE) = Ji(BnE) = 0. 

Since tt and the maximal irreducibility measure fx are equivalent, tt admits a density on E. 
Now, we investigate the expression of this density. By Fubini's theorem, we have for any 
B c E, 

tt(B) = f_n(dy)n(y,B) 
Je 

7r(dy) / r(y, x)dx 
Jb 

ix{dy)r{y,x) ) dx, 



As a consequence, one may identify p with the function / ir(dy)r(y, •). □ 

Je 

One shall see that the regularity of the conditional probability density function r is signif- 
icant in all the sequel. We state that under additional assumptions, r is Lipschitz. 

Assumptions 3.7. We assume the following statements. 
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(i) t~ is a bounded and Lip schitz function, that is, 

^[t~]u v > 0, Mx,y^E, \t~(x) - t~(y)\ < [t~] Lip \x - y\. 
(ii) The flow $ is Lipschitz, that is, 

3[3?]z,ip > 0, Vx, y G R d , Vt G R, | $ x (i) - $„(t)| < [$\ L i P \x ~ y\- 
(Hi) f is a bounded and Lipschitz function, that is, 

3[f] Lip >0, Vx,y£R d , Vi G R, \f(x,t)-f(y,t)\ < [f] Lip \x - y\. 
(iv) q is a bounded and Lipschitz function, that is, 

3[q]u P > 0, Vx,y,z G R d , \q(x,y) - q(x, z)\ < [q]u P \y ~ z\. 
(v) is a bounded and Lipschitz function, that is, 

3[D$] Lip > 0, \/x, y G R d , Vt G R, - | < [D^} Lip \x - y\. 

Proposition 3.8. r is a bounded function. Furthermore, there exists a constant [r]iip > 
such that, for any x G E, y G E and u G R d such that y + u G E , we have 

\r(x,y + u) - r(x,y)\ < [r] Lip \u\. 
Proof. First, we have from (9), 



< II* Hoc [M| DO [|/||oo||-D$[|oo- 

For the second point, we consider the function 7 defined by, 

V(!/,t) G K d x R, 7 (y,t) = gfo^C-tJJ/C^C-t^D^C-t). 

This function is Lipschitz as a compound and product of Lipschitz functions (see Assump- 
tions 3.7). [^}lip stands for its Lipschitz constant, and we have 

\l(y,t) ~lf(y + u,i)\ <[j]u P \u\. 

In addition, by (9), the function r(x,y) is given by 

f-t~(y) 

r(x,y) = / 7 (y,s)ds. 



We suppose that —t~(y) < —t~{y + u) (recall that t~ is a negative function). We have 

r-t~(y) r-t.-(y+u) 

r(x,y + u) -r(x,y) = (j(y + u, s) - j(y, s)) ds + / -f(y + u,s)ds. 

Jo J-t-(y) 

As a consequence, 

f'\\t 1 1 00 

\r(x,y + u)-r(x,y)\ < / | 7 (y + u, s) - 7 (y, s)| ds 

J 

+ Ikllooll/llooll^lloo |i~(y)-r(y + «)| 

< ||<"||oo[7]iipl w l + [M|oo[|/[|oo[|£*||ooM- 

The obtained inequality for —t~{y) > —t~{y + u) is exactly the same one. This achieves 
the proof. □ 
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3.2. Estimation of p. We propose a recursive nonparametric estimator of the function p 
given in the Corollary 3.6. In all the sequel, we consider a kernel function K which satisfies 
the following assumptions. 

Assumptions 3.9. The kernel K : R d — > R + satisfies: 

(i) supp.fr C i?(0 R d, 5), where 5 > 0. B(x, r) stands for the open ball centered at x with 

radius r. 
(ii) K is a bounded function. 

Under Assumptions 3.9, J* Rd K 2 (y)dy is finite, r 2 denotes this integral in the sequel. For 
all integer n, the recursive estimator p n of p that we propose is given for all x £ E by 

1 £1 i _(zr- x \ 



(13) Pn{x) = - Y,~d K 



3=1 3 \ J / 

where the bandwith Vj satisfies 

vj = vij~ a , with a > 0. 
Remark 3.10. Let x £ E and j > 1. Since the sequence (v n ) is decreasing, we have 

suppA( ) CsuppiCf — J C B(x,viS). 

V Vj ) \ vi J 



Thus, if V\S < dist(x,dE), we have 



x 



supp K C E. 

In the following proposition, we establish the pointwise asymptotic consistency of p n . 

Proposition 3.11. Let x E E. One chooses v% such that v\5 < dist(x, dE) and a such 
that ad < 1. Then, 

p n {x) ^p(x), 

when n goes to infinity. 



Proof. By the expression of p(x) given by (12), the difference p n (x) —p(x) may be written 
in the following way, 

1 »±1 j (ZT-x\ f 
Pn(x)-p(x) = -J>-d K \ ] - r(u,x)ir(du) 

(14) = \ K ( Z ^-) + l -M n + R^ + R^, 

nvf \ v\ J n 
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where M n , Rn^ and Rn are given by 



Mr, 



3=1 



-K 



Z j+i ~ x 



R-' 



r[Z- ,x + yv j+1 )K(y)dy 



(i) 



R 



R^ 



3=1 



~E / r ( Z i > x + V v i+i) ~ r ( Z i > x ) K (y)dVi 
n i ./Rd L J -I 



1 n 

- - J_r(u,x)ir(du). 



The dependency on x is implicit. In (14), the first term clearly tends to as n goes to 
infinity. The sequel of the proof is divided into three parts: in the first one, we show that 
Rn^ tends to by the ergodic theorem. In the second one, we focns on J#> and we prove 
that this term goes to 0. Finally, we state that M n /n tends to by using the second law 
of large numbers for martingales. 

Recall that the Markov chain (Z~) is positive Harris-recurrent with invariant measure tt, 
according to Proposition 3.3. Thus, one may apply the ergodic theorem (see for instance 

Theorem 17.1.7 of [IS]) and we obtain that R^ almost surely tends to 0. For Rn , we 
have 

rt 



1 



I f 

- ~E / r(Z~ x + yv j+1 )-r(Zr,x)K(y)dy 
r ]Li P \y\vj+iK(y)dy 
y\K(y)dy [r] 

Lip- 



< 



< 



1 

-E 

3=1 



This upper bound tends to by Cesaro's lemma because the limit of the sequence (v n ) is 
0. Therefore, R^ goes to as n tends to infinity. Finally, we investigate the term M n /n. 
First, we show that the process (M n ) is a discrete-time martingale with respect to the 
filtration (J- n ) defined by, 



Vn > 1, T n = o{Z x ,...,Z n , Z n+l ). 



We have 
E [M n \ T n -x] = M n _i + E 



K 



IJ n+l 



Z n+l ~ 00 
V n +1 



z: 



r(Z n ,x + yv n+1 )K(y)dy. 



Thus, we only have to prove that 
1 



(15) 
We have 



E 



-K 



J n+l 



Zn+l X 
V n +1 



z: 



r(Z n , x + yv n+ i)K(y)dy. 
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By the assumption on v\ and Remark 3.10, 



V n +1 

R ,u-x 

E V v n+l 



n(z n , du) 

r(Z~,u)du, 



by (8). Finally, the change of variable u = yv n+ \ + x states (15). Thus, (M n ) is a 
martingale. We shall study the asymptotic behavior of its predictable quadratic variation 
< M >. A straightforward calculus leads to 



(M n - M n _i) 



1 



2d 
u n+l 



-K 1 



Z n+1 ~ 00 
V n +1 



+ 



1 2 



J n+1 



-K 



Z n+l ~ X 
V n +1 



r(Z n ,x + yv n +i)K(y)dy 
r(Z~ , x + yv n+1 )K(y)dy. 



Using the method used to show (15), we deduce that 

E [(M n - M n -i) 2 \T n ] = [ K 2 (y)r(Z- , x + yv n+1 )dy 

V n+1 JE 



L JE 



r(Z n ,x + yv n+1 )K(y)dy 



As a consequence, 



n ( 1 

< M >n</2[^d— 
3 =1 \ V J+1 



\r\\ooT + \\r\ 



est n 



ad+l 



as n tends to infinity. By the second law of large numbers for martingales (see Theorem 
1.3.15 of [9]), we have 



Ml = O (< M > n ln(< M >n) 1+7 ) , with 7 > 0. 



As a consequence, 



M, 
n 



-=0 (^Jn ad - 1 ln(n ad + 1 ) 1 +^ . 



Thus, M n /n almost surely tends to as n goes to infinity if ad < 1. This achieves the 
proof. □ 



4. Estimation of the invariant distribution of (Z~ , Z n ) 

In this section, we state that the Markov chain (Z~ , Z n ) admits a unique invariant 
measure. In addition, we are interested in the recursive estimation of this measure. 



4.1. Some properties of (Z n ,Z n ). In this part, we focus on the asymptotic behavior 
of the chain (Z~,Z n )- In all the sequel, r] n (respectively TT n ) denotes the distribution of 
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(Z n , Z n ) (resp. Z n ) for all integer n. We have these straightforward relations between r/ n , 
Q or q and 7r n , 

7] n (A X B) = [ Q(z,B)7T n (dz) 
J A 

(16) = / q(z,y)TT n (dz)dy. 

Lemma 4.1. We have 



AxB 



lim \\rj n - rj\\ T v = 0, 

n— >+oo 

where the limit distribution r) is defined for all A x B € B(E x E?) by 

(17) i)(ix5) = / q(z,y)ir(dz)dy. 

J AxB 

Proof. Let 5 be a measurable function bounded by 1. By virtue of Fubini's theorem, we 
have 



L 



9{x, y) (r] n (dx x dy) - r](dx x dy)) 



ExE 

from (16) and (17). Thus, 



< 



(vr n (dx) - vr(dx)) / g(x,y)q(x,y)dy 
E Je 



g(x,y) (r] n (dx x dy) - r](dx x dy)) 



ExE 



< 



g(x)(ir n (dx) - n(dx)) 



where the function g : x 1— > f E g(x,y)q(x,y)dy is bounded by 1 since g is bounded by 1 
and q is the conditional density associated with the Markov kernel Q. As a consequence, 



( 18 ) \\Vn ~ v\\tV - II 71 "™ ~ HItV • 

One obtains the expected limit from (6). 

In addition, one may prove that r\ admits a density on E x E. 
Lemma 4.2. There exists a positive function h such that 



□ 



7](A x B) = I h(x,y)dx dy, 
J AxB 

for any A x B G £>(£? x E 1 ) with Ad E. In addition, h is given for all x,y £ E by 
(19) h(x,y) = p(x)q(x,y). 



Proof. From (17), we have 



r](A x B) 



q(z,y)7r(dz)dy 



AxB 



q(z,y)p(z)dzdy, 



AxB 



by (11) and because A C E. This achieves the proof. 



□ 
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4.2. Estimation of h. We propose to estimate the function h by the recursive nonpara- 
metric estimator h n given, for any (x,y) £ E 2 , by 

where the bandwith Wj is given by 

Wj = wij~P , with (3 > 0. 

The kernel function K is assumed to satisfy Assumptions 3.9. In the sequel, we are 
interested in the pointwise convergence of the estimator at a point (x, y) EE 2 . We assume 
that wi is such that w±5 < dist(x, dE), where 5 is the radius of the open ball which contains 
the support of the kernel function K. In this case, Remark 3.10 is still valid, and we have 
the following inclusions, for any integer j, 

(21) suppK ( — - ) C B(x, Wl 5) C E. 

V Wj ) 

Our main objective is to state in Proposition 4.7 that h n (x,y) almost surely converges 
to h(x,y). First, we show that this estimator is asymptotically unbiased (see Proposition 
4.4). 

We state some new properties of the distribution measures 7r n and it. Let us recall that 
7r n is the law of Z~, while n is the invariant measure of the Markov chain (Z~). 

Lemma 4.3. We have the following statements. 

• For any integer n, 7r n admits a density function p n on E. 

• p n is bounded by \\r\loo and is an [r]Li p -Lipschitz function. 

• p is Lipschitz. 

• For any integer n, we have 

(22) sup \ Pn {x) - p(x)\ < WrW^Kp-^-V. 
Proof. For the first point, let B £ B(E) with B C E. We have 

MB) =\l n{tAvW-Mi) 

Je J b 

r(€,y)TTn-i(dQdy, 




B JE 



where r is the conditional density associated with the kernel 1Z (see (8)). Thus, one may 
identify 

(23) Pn (y) = / r(£,j/)7r n _i(d£). 

Je 

For the second assertion, we have stated in Proposition 3.8 that r is a bounded function. 
As a consequence, 

\Pn(y)\ < \\r\\ oc TT n ^ 1 (E) = \\r\loo. 
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In addition, since r is Lipschitz, 

\Pn(y) - Pn(z)\ < I \r{£,y)-r(€,z)\Tr n -i(dg) 



< 



[r]Li P \y - z\-K n -i(E) 



[r]u P \y ~ z\ 



For the third point, p is Lipschitz for the same reason than p n since p satisfies (12). Finally, 
for the last point, we have, by (12) and (23), 

\Pn{x)-p{x)\ < lr(y,x)\Tr n -i(dy)-TT(dy)\ 
Je 



by (6). This achieves the proof. 



□ 



Now, one may state that h n (x,y) is an asymptotically unbiased estimator of h(x,y). 
Proposition 4.4. When n goes to infinity, 



E 



h n (x,y) -+h(x,y). 



Proof. We only state that 



E 



h n (x,y) 



n + 1 



n 



h(x,y) 



tends to 0. We have 
n + l 



E 



h n (x,y) 



n+l 



n 



ExE W 



l T ^(u — x\ /v — y. 
jK | I K | I TTj(du)q{u,v)dv 



2d 



W ; 



h(x, y)K(u)K{v)dudv 



ExE 



Thanks to (21), one may replace E by E in the first integral. As a consequence, one may 
replace Wj(du) by pj(u)du (see Lemma 4.3). Together with (19) and a change of variables, 
we obtain 



E 



h n (.x,y) \ - r ^^h{x,y) 



n 



- n+l „ 

= — 2 / K(u)K(v)(pj(x + uvjj)q(x + uwj,y + vvjj) — p(x)q(x,y))dudv. 
n ~ v JExE ^ ' ' 

Furthermore, since pj is [rJ^p-Lipschitz and bounded by HrHoo in the light of Lemma 4.3, 
an elementary calculus leads to 

\pj(x + uwj) q(x + uwj,y + vwj) — p(x)q(x,y)\ 

< \\ r \\oo[q}Lip\v\wj + Halloo [r] Lip |w|u;j + WqW^ \pj(x) -p(x)\ 



< wj (\\rWoolq\up\v\ + ||g||ooHiip|«|) + \\q\\ 



-Cj-i) 
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with (22). Finally, we obtain 
E 



h n (x,y) -^-^l/i(x,y) 
n 



< (lHloo[g]l*P + IMIooMLip) f E K(u)\u\du | ||r||oo||g||ooK y> 

3=1 3=1 
, (IH|oo[g]z,ip + l|g||ooMLip) J E K(u)\u\du ^ |M|oo||g||ooK 

" ~ n hi" nil-p- 1 ) ' 

which tends to by Cesaro's lemma. □ 

In the following, we are interested in some properties of the discrete-time process 

which naturally appears in the study of the estimator h n (x,y). In particular, we propose 
to investigate its autocovariance function. On the strength of this result, we will establish 
the asymptotic behavior of the variance of h n (x,y). 

Proposition 4.5. There exist two constants B and b > 1 such that, for any integers n > k, 

(25) \Cov(A k ,A n )\ < \M^E b k-n ( l + h - 

w* d V 

In particular, one obtains by taking k = n and using that b~ n < 1, 

(26) Var(^ n ) < 

w 

Proof. We have 
Cov(A k ,A n ) 



Ull d wl d Vll^ll^o V w k J V W k J ' \\K\\la \ W n J V w 

where both the components in the covariance are bounded by 1. We apply Theorem 16.1.5 
of [18] with V = 1, 3> = (Z~,Z), 

9 = tJ^k(—)k( : ^ L ) and h = J^K ( ^) K <' ~ y 



\ K \\lo V ™n J V w n J \\ K WL V w k J \ u>k 

The conditions of the theorem are satisfied by (6) and (18). We obtain 

\Cov(A k ,A n )\ < M^Bb k - n (l + b- k 



w% d wl d 



Together with l/wfr < l/w^ , this shows (25). □ 

In the following result, we give a bound for the variance of h n (x,y). It is a corollary of 
Proposition 4.5. 
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Corollary 4.6. Let n be an integer. We have 

8 WKW^B 



(27) Var [h n (x,y)\ < . 

As a consequence, this variance goes to when Af3d < 1 (recall that w n = w\n~^ ). 

Proof. This inequality is a consequence of (25) stated in Proposition 4.5. Indeed, in light 
of the expressions of A n (24) and h n {x,y) (20), we have 

^ n+ln+l 

Var(X(x,y)J = -^^^Cov^^fc) 

k=l l=k 
„ n+ln+l 

Using that b~ k < 1 and w^ M < w~$, 

n+1 n+1 



W, 



Var < -Ar\\K\\LBj2b k ^ b ' 1 

71 W n+l k =X l=k 

" <+i ^ 1-6 1 

< 8 II^H^ 
~ «<i 1 - 6" 1 ' 

with (ra + l)/n < 2. □ 
Now, one may state the consistency of our estimator of h{x, y). 

Proposition 4.7. Let (x,y) G E 2 . One chooses wi5 < dist(x, dE) and &fid < 1. Then, 

h n (x,y) ^ h(x,y), 

when n goes to infinity. 

Proof. According to Proposition 4.4, we only have to prove that 
(28) Y n = (h n (x,y)-n\7i n (x,y)jy 

almost surely converges to 0. In the sequel of the proof, we establish that there exists a 
random variable Y such that Y n Y. Since the sequence (Y n ) tends to in L 1 (remark 
that E[Y„] = Var (h n (x,y)\, together with Corollary 4.6), Y = a.s. and it induces the 

expected result. In order to show the almost sure convergence of the sequence (Y n ), we 
use Van Ryzin's lemma (see [21]). In light of this result, if the sequence (Y n ) satisfies the 
following conditions, 

(i) Y n > a.s., 

(ii) E[Yi] < +oo, 

(iii) E |«S n ] < Y n + Y^ a.s., where S n = o~(Y\, . . . , Y n ) and Y^ is immeasurable, 

(iv) En>iE[|KI]<+oo, 
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then Y n ^ Y. In our context, points (i) and (ii) are unquestionably satisfied. By (20), 
(24) and (28), we have 

-. n+l 



k=l 



Consequently, we have the recurrence relation, 

A n+l - E[A n+i ] + (n - 1)V%^I 



By squaring, we obtain 

2 



n - 1 \ * (A w+1 -EK +1 ]) z + 2(n - 1)^^7(^+1 - E[A n+1 ]) 

-in— 1 i 



< 5^-1 + 



(A n+1 - E[^l n+1 ]) 2 + 2(n - l)yT^IT(A n+1 - E[A n+ i]) 



n- 



Finally, E[y n |5 n _i] < Y n _i + where 



(29) y n _! 



rE 



71" 



(Ah-1 " EK+i]) 2 + 2(n - 1)^(4+1 - EK + i]) 



<5 n -i 



Thus, (hi) is checked. Ultimately, we have to verify (iv). By (29) and Cauchy-Schwarz 
inequality, 



E [l y n-ll] < ^Var(AH-l) + 2(n n 2 [0^1 \ A n+l ~ E [^+l 

< -^Var(A-fl) + -VE[r n _i]Var(^ n +i) 



< -^Var(A n+1 ) + Vvar (X_i(:c, y)) Var(^„ +1 ). 
Thus, by (26) and (27), there exist two numbers ci and C2 such that 



< 



n 2 wff n 3 / 2 w^ d 
ci c 2 



+ 



As a consequence, 5^E[|Y^|] is a convergent series for 8/3cZ < 1. 



□ 



Our main convergence result is stated in the following theorem. The recursive estimator 
of q(x, y) that we consider may be written as follows, 



n+l 



y 1 K 5 *\ K ( z i 



1 w , 



W; 



where the couple (x, y) is in E 2 , Vj = v\j a , Wj = w\j with a, (3 > 0, and fT is a kernel 
function satisfying Assumptions 3.9. 
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Theorem 4.8. Let us choose v\ and w\ such that max(i>i, wi)5 < dist(x, dE). Ifp(x) > 0, 
ad < 1 and 8f3d < 1, then, when n goes to infinity, 

q n {x,y) ^ q{x,y). 

Proof. In light of (19), if p(x) > 0, one may write q(x,y) = h(x,y)/p(x). In addition, 
q n (x,y) is defined by the ratio h n (x,y)/p n (x) (see (20) for the expression of h n (x,y) and 
(13) for the one of p n (x)), where h n (x,y) (respectively p n {x)) estimates h(x,y) (resp. 
p(x)). In such a case, the result is a corollary of Propositions 3.11 and 4.7. □ 

5. Simulation study 

The goal of this section is to illustrate the asymptotic behavior of our recursive estimator 
via numerical experiments in the one-dimensional case. More precisely, we investigate 
numerical simulations for a quite simple example in Subsection 5.1, and for the well-known 
TCP window size process in Subsection 5.2. 

5.1. First numerical experiments. We consider a PDMP (X t ) defined on the state 
space E =]0, 1[, and we assume that the process starts from the midpoint of E, Xq = 0.5. 
The main characteristics of (Xt) are defined as follows. The flow $ satisfies $ x (t) = x + 1 
for any x £ E and t £ R. In addition, the jump rate A is chosen constant equal to 10. 
Finally, for x £ E, Q(x, •) is the truncated exponential distribution with mean 1/(10 + x). 
All the conditions assumed in the present paper are obviously satisfied for this example. 
In particular, the transition kernel Q admits a density q with respect to the Lebesgue 
measure. 
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FIGURE 1. Boxplots, over 100 replicates, of the estimation of q(x,y), with 
x = 0.2 and y = 0.6, from different numbers of observed jumps. When 
Pn(x) = 0, we replace q n (x,y) by 0. As a consequence, they are lots of 
zeros over the 100 replicates when n is too small. 
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In the sequel, we focus our attention on the nonparametric estimation of the conditional 
density q associated with Q. For any couple (x,y) £ E 2 , the function of interest q(x,y) 
satisfies 

q(x,y) = — (10 + x)exp(-(10 + x)y), 

where K x stands for the normalizing constant. We choose the Epanechnikov kernel K 
given, for any x 6 R, by 

3 

K (x) = -(1 - x 2 )l Ux \< 1} , 

which satisfies all the assumptions imposed in the theoretical part of the paper. Finally, 
we select the constant parameters a and /3 such that 

a = 8 = 1 - 10~ 2 . 

H 8 

Figure 1 presents the empirical distribution over 100 replicates of the estimate q n (x,y), 
with x = 0.2 and y = 0.6, from different numbers of observed jumps. On small-sampled 
sizes, our procedure is quite unfulfilling. However, for n large enough, our method succeeds 
in the estimation of the quantity of interest, especially when n is greater than 10000. In 
addition, the complete curve q(x, y), with x = 0.5 and 0.1 < y < 0.9, and its estimate from 
20000 jumps are very close (see Figure 2). The estimation is provided only on the interval 
[0.1,0.9] C E, since the almost sure convergence of q n has been stated on the interior of 
the state space. 




0.2 0.4 0.6 0.8 



State space 

Figure 2. Estimation of q(x,y), x = 0.5, for 0.1 < y < 0.9 from 20000 
observed jumps. The real function is drawn in dashed line, while its estimate 
is in solid line. 

5.2. The TCP window size process. The TCP window size process naturally appears 
in the modeling of the famous Transmission Control Protocol used for data transmission 
over the Internet. For this special case of PDMP, the sample paths are piecewise-linear 
and the whole randomness of the dynamics comes from the jump times. The motion of 
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the TCP window size process (Xt) on E =]0, +oo[ is governed by its features <I>, A and Q. 
The flow $ is linear, that is, & x (t) = x + t for any x G E and i 6 R. The jump rate A 
satisfies A(x) = x. Finally, the transition kernel Q is given, for any couple (x,y) G E 2 , by 
Q(x,dy) = <5{x/2}(dy)- One may refer the interested reader to [4] for a study on the long 
time behavior of this process. 




FIGURE 3. Estimation of Q(x,dy) = 6f x /2\(dy), with x = 0.5, from dif- 
ferent numbers of observed jumps. The vertical line stands for the value 
x/2 = 0.25. 

As mentioned before, the transition kernel Q does not admit a density with respect to 
the Lebesgue measure. As a consequence, our approach is theoretically not well-adapted 
to tackle the estimation of Q. Nevertheless, we apply our estimator in this unsuitable 
framework in order to analyze its asymptotic behavior. Despite this weakness, one may 
observe on Figure 3 that our method performs pretty well. The kernel function and the 
constants a and (3 have been chosen in the same way as in the previous example. 

Acknowledgments. The author would like to thank his two PhD advisors, Frangois 
Dufour and Anne Gegout-Petit, for their suggestions and helpful comments. 
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